The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations

Background: Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.Methods: a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.Results: The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.Conclusions: An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use

Changes in standard of candidates taking the MRCP(UK) Part 1 examination, 1985 to 2002: Analysis of marker questions

The maintenance of standards is a problem for postgraduate medical examinations, particularly if they use norm-referencing as the sole method of standard setting. In each of its diets, the MRCP(UK) Part 1 Examination includes a number of marker questions, which are unchanged from their use in a previous diet. This paper describes two complementary studies of marker questions for 52 diets of the MRCP(UK) Part 1 Examination over the years 1985 to 2001 to assess whether standards have changed

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies

Background: Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applicant abilities. Methods: Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register. Construct-level predictive validity was calculated with the method of Hunter, Schmidt and Le (2006), adapted to correct for right-censorship of examination results due to grade inflation. Results: Meta-regression analyzed 57 separate predictor-outcome correlations (POCs) and construct-level predictive validities (CLPVs). Mean CLPVs are substantially higher (.450) than mean POCs (.171). Mean CLPVs for first-year examinations, were high for A-levels (.809; CI: .501 to .935), and lower for GCSEs/O-levels (.332; CI: .024 to .583) and UKCAT (mean = .245; CI: .207 to .276). A-levels had higher CLPVs for all undergraduate and postgraduate assessments than did GCSEs/O-levels and intellectual aptitude tests. CLPVs of educational attainment measures decline somewhat during training, but continue to predict postgraduate performance. Intellectual aptitude tests have lower CLPVs than A-levels or GCSEs/O-levels. Conclusions: Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance (and measurement error, range restriction and right-censorship have been taken into account). Just as in astrophysics, ‘dark matter’ and ‘dark energy’ are posited to balance various theoretical equations, so medical student selection must also have its ‘dark variance’, whose nature is not yet properly characterized, but explains a third of the variation in performance during training. Some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills

Measuring Outcome after Wrist Injury: Translation and Validation of the Swedish Version of the Patient-Rated Wrist Evaluation (PRWE-Swe)

<p>Abstract</p> <p>Background</p> <p>There is a need for outcome measurement instruments for evaluation of disability after trauma. The Patient-Rated Wrist Evaluation (PRWE) is a self-administered region-specific outcome measuring instrument developed for use in evaluating disability and pain of the wrist. The aim of this study is to translate and to cross-culturally adapt the PRWE for use in a Swedish patient population. Moreover, we aim at investigating the PRWE in terms of validity, reliability and responsiveness.</p> <p>Methods</p> <p>We performed a translation and cross-cultural adaptation of the PRWE to Swedish (PRWE-Swe), utilising the process recommended by the American Association of Orthopedic Surgeons. A total of 124 patients with an injury to the wrist were included in the study. They filled in the PRWE and the DASH questionnaires at two separate occasions.</p> <p>Results</p> <p>Reliability of the PRWE in terms of internal consistency (Cronbach's alpha 0.97) and test-retest stability (intraclass correlation coefficient 0.93) were excellent. Face validity and content validity were judged as good. Criterion validity assessed as the correlation between the PRWE and the DASH was also good (Spearman's rho = 0.9). Responsiveness measured by the standardized response mean (SRM) was good with an SRM_PRWEof 1.29.</p> <p>Conclusion</p> <p>This Swedish version of the PRWE is a short and easily understood self-administered questionnaire with good validity, reliability, and responsiveness. Our results confirm that the PRWE is a valuable tool in evaluating the results after treatment of a wrist injury.</p

A self-rating scale for patient-perceived side effects of inhaled corticosteroids

BACKGROUND: Patient-reported side effect questionnaires offer a simple method for the systematic measurement of drug-related side effects. In order to measure patients' inhaled corticosteroids (ICS) related side effect perceptions the 14-day retrospective Inhaled Corticosteroid Questionnaire (ICQ) was developed. In this research we aim to assess the construct validity and reliability of the ICQ and test its responsiveness to dose changes in adult asthma patients. METHODS: In a cross-sectional study, current inhaler users with asthma completed the ICQ (27 with non ICS inhaler; 61 BDP equivalent daily ICS low dose ≤400 μg; 62 mid dose 401–800 μg; and 105 with high dose >800 μg). We generated 3 construct validity hypotheses: 1) a hierarchical dose-response pattern for scoring of the individual items on the ICQ, and statistically significant differences in the scores of each of the 15 ICQ domains by ICS dose group 2) an association between ICS dose and ICQ scoring after adjusting for appropriate confounders in multiple regression; 3) greater convergence between local side effect domains than between systemic and local domains of the scale. Test-retest reliability was assessed on a randomly selected subgroup of patients (n = 73) who also completed the ICQ a second time after 7 days. In a separate longitudinal study, 61 patients with asthma completed the ICQ at baseline and after changing their daily ICS dose, at 2- and 6- months, in order to test the ICQ's responsiveness. RESULTS: All three construct validity hypotheses were well supported: 1) a statistically significant difference existed in scores for 14 domains, the high ICS dose group scoring highest; 2) ICS dose independently predicted ICQ scoring after adjusting for confounders; 3) greater convergence existed between local ICQ domains than between local and systemic domains. The ICQ had good reproducibility: test-retest intraclass correlation coefficients were ≥0.69 for all but the 'Facial Oedema' domain. In the longitudinal study, ICQ scores for 'Voice Problems' changed significantly at 2- and 6-months from baseline and other ICQ domains displayed trends in scoring change accordant with dose modulation at 6-months. CONCLUSION: The ICQ has good dose-related discriminative properties, is valid, reliable, and shows potential responsiveness to ICS dose change

Alcoholic beverages and risk of renal cell cancer

Using a mailed questionnaire, we investigated the risk of renal cell cancer in relation to different types of alcoholic beverages, and to total ethanol in a large population-based case–control study among Swedish adults, including 855 cases and 1204 controls. Compared to non-drinkers, a total ethanol intake of >620 g month−1 was significantly related to a decreased risk of renal cell cancer (odds ratio (OR) 0.6, 95% confidence interval (CI) 0.4–0.9; P-value for trend=0.03). The risk decreased 30–40% with drinking more than two glasses per week of red wine (OR 0.6, 95% CI 0.4–0.9), white wine (OR 0.7, 95% CI 0.4–1.0), or strong beer (OR 0.6, 95% CI 0.4–1.0); there was a clear linear trend of decreasing risk with increasing consumption of these beverages (P-values for trends <0.05)

Deprived or not deprived? Comparing the measured extent of material deprivation using the UK government's and the Poverty and Social Exclusion surveys' method of calculating material deprivation

Poverty can either be measured directly, through standards of living such as material deprivation, or indirectly through resources available, usually income. Research shows that the optimum measure of poverty combines these methods, a fact that the UK government took cognisance of in its tripartite measure of child poverty. For use in a birth cohort study, two methods of calculating material deprivation were tested: the method used by the UK government taken from the Family Resources Survey (FRS), and the methods used in the Poverty and Social Exclusion (PSE) study at Bristol University. Results show that the former measure, compared to the latter measure, underestimates the depth and extent of material deprivation among families with young children in Scotland

Doubly Uniparental Inheritance of Mitochondria As a Model System for Studying Germ Line Formation

BACKGROUND: Doubly Uniparental Inheritance (DUI) of mitochondria occurs when both mothers and fathers are capable of transmitting mitochondria to their offspring, in contrast to the typical Strictly Maternal Inheritance (SMI). DUI was found in some bivalve molluscs, in which two mitochondrial genomes are inherited, one through eggs, the other through sperm. During male embryo development, spermatozoon mitochondria aggregate in proximity of the first cleavage furrow and end up in the primordial germ cells, while they are dispersed in female embryos. METHODOLOGY/PRINCIPAL FINDINGS: We used MitoTracker, microtubule staining and transmission electron microscopy to examine the mechanisms of this unusual distribution of sperm mitochondria in the DUI species Ruditapes philippinarum. Our results suggest that in male embryos the midbody deriving from the mitotic spindle of the first division concurs in positioning the aggregate of sperm mitochondria. Furthermore, an immunocytochemical analysis showed that the germ line determinant Vasa segregates close to the first cleavage furrow. CONCLUSIONS/SIGNIFICANCE: In DUI male embryos, spermatozoon mitochondria aggregate in a stable area on the animal-vegetal axis: in organisms with spiral segmentation this zone is not involved in cleavage, so the aggregation is maintained. Moreover, sperm mitochondria reach the same embryonic area in which also germ plasm is transferred. In 2-blastomere embryos, the segregation of sperm mitochondria in the same region with Vasa suggests their contribution in male germ line formation. In DUI male embryos, M-type mitochondria must be recognized by egg factors to be actively transferred in the germ line, where they become dominant replacing the Balbiani body mitochondria. The typical features of germ line assembly point to a common biological mechanism shared by DUI and SMI organisms. Although the molecular dynamics of the segregation of sperm mitochondria in DUI species are unknown, they could be a variation of the mechanism regulating the mitochondrial bottleneck in all metazoans

Flavonoids uptake and their effect on cell cycle of human colon adenocarcinoma cells (Caco2)

Green tea, mainly through its constituents epigallocatechin gallate, epigallocatechin, epicatechin gallate and epicatechin, has demonstrated anticarcinogenic activity in several animal models, including those for skin, lung and gastro-intestinal tract cancer, although less is known about colorectal cancer. Quercetin, the major flavonoid present in vegetables and fruit, exerts potential anticarcinogenic effects in animal models and cell cultures, but less is known about quercetin glucosides. The objectives of this study were to investigate (i) the antioxidant activity of the phenolic compounds epicatechin, epigallocatechin gallate, gallic acid and quercetin-3-glucoside; (ii) the cytotoxicity of different concentrations of epicatechin, epigallocatechin gallate, and gallic acid; (iii) the cellular uptake of epicatechin, epigallocatechin gallate, gallic acid and quercetin-3-glucoside and (iv) their effect on the cell cycle. Human colon adenocarcinoma cells were used as experimental model. The results of this study indicate that all dietary flavonoids studied (epicatechin, epigallocatechin gallate, gallic acid and quercetin-3-glucoside) show a significant antioxidant effect in a chemical model system, but only epigallocatechin gallate or gallic acid are able to interfere with the cell cycle in Caco2 cell lines. These data suggest that the antioxidant activity of flavonoids is not related to the inhibition of cellular growth. From a structural point of view, the galloyl moiety appears to be required for both the antioxidant and the antiproliferative effects